fix(goose): read token totals from accumulated_usage on goose 1.39+ - #106

Fixes #74. This is assigned to me.

Note for reviewers: the commits in this PR do not have Entire-Checkpoint trailers. I know CONTRIBUTING.md asks for them. The Entire CLI was not capturing the session this was written in, and I did not want to add trailers that don't point at a real checkpoint. If you need them before merge, tell me and I'll redo the commits with Entire capturing. I also could not get the Copilot reviewer response CONTRIBUTING mentions (my account is out of quota). I ran Greptile, Qodo, Kilo and Gitar on a fork PR first and addressed their findings.

Why

Upstream issue 74: on current goose, entire-agent-goose calculate-tokens reports zero tokens for every session. The adapter reads the flat accumulated_input_tokens / accumulated_output_tokens export fields. goose stopped writing them in 1.39.0, where Session gained nested usage and accumulated_usage objects instead. Nothing errors, so the zeros go unnoticed.

What

• CalculateTokens now checks the token shapes in this order: accumulated_usage (the whole session, compacted history included), then usage (the current context window), then the old flat fields for goose 1.38 and earlier.
• A nested object counts as present only when it carries an input_tokens or output_tokens value (a reported 0 counts). A missing object, goose's all-null default Usage, or one holding only total_tokens/cache figures falls through to the next shape. Fields are never mixed between accumulated_usage and usage, because they cover different spans (whole session vs current context window).
• Nested exports now give a cache split: cache_read_input_tokens → cache_read_tokens and cache_write_input_tokens → cache_creation_tokens.
• Different from the issue's expected output: goose documents Usage.input_tokens as all prompt tokens, cache reads and writes included (token_usage.rs, 1.39.0+). Entire's InputTokens means fresh input only, and Entire adds it to the cache fields. To avoid counting cache twice, input_tokens is reported as input − cache_read − cache_write, never below 0. Using the issue's 1.46.0 numbers that gives in=14, not 124611. As a check, 14 + 112820 + 11777 + 394 = 125005, which matches goose's own total_tokens.
• usage and accumulated_usage are added to the session header that gets stamped onto every stored JSONL record. Entire runs calculate-tokens on the stored transcript, sometimes on a line-sliced part of it, so the totals have to survive that.
• AGENT.md and README now describe the new token fields.

Tests (new tokens_test.go)

Cases: accumulated_usage preferred over usage; usage used when accumulated_usage is all null; usage used when accumulated_usage is absent; pre-1.39 flat fields; null cache fields; legitimately zero totals; no token fields at all; cache larger than input (input stays at 0); nested totals surviving the stored JSONL path and a scoped slice.

Before (new tests run against unchanged main):

--- FAIL: TestCalculateTokensExportShapes/accumulated_usage_is_preferred_over_usage
    tokens = {InputTokens:0 CacheCreationTokens:0 CacheReadTokens:0 OutputTokens:0 APICallCount:1},
    want    {InputTokens:14 CacheCreationTokens:11777 CacheReadTokens:112820 OutputTokens:394 APICallCount:1}
--- FAIL: .../usage_is_used_when_accumulated_usage_is_all_null
--- FAIL: .../usage_is_used_when_accumulated_usage_is_absent
--- PASS: .../pre-1.39_flat_accumulated_fields_still_read
--- FAIL: .../null_cache_fields_count_as_no_cache
--- PASS: .../legitimately_zero_accumulated_usage_stays_zero
--- PASS: .../no_token_fields_at_all_reports_zero_without_error
--- FAIL: .../cache_larger_than_input_never_yields_negative_fresh_input
--- FAIL: TestCalculateTokensNestedUsageSurvivesStoredTranscript/full_stored_transcript
--- FAIL: TestCalculateTokensNestedUsageSurvivesStoredTranscript/scoped_slice
FAIL

(The three cases that pass on main are regression guards: the legacy shape and the zero cases, which main already handles correctly.)

After: all 10 subtests and the existing TestCalculateTokens pass.

Binary smoke test, calculate-tokens --offset 0 on a 1.39+-shaped export:

main: {"input_tokens":0,"cache_creation_tokens":0,"cache_read_tokens":0,"output_tokens":0,"api_call_count":1}
fix:  {"input_tokens":14,"cache_creation_tokens":11777,"cache_read_tokens":112820,"output_tokens":394,"api_call_count":1}

Checks (Go 1.26.0, golangci-lint 2.11.3; mise was not available, so I ran the commands mise would run): gofmt -s -l . clean; golangci-lint run reports 0 issues in all 9 Go modules; go test ./... passes for every Go agent; go test -race passes for goose; every Go agent builds; go vet -tags=e2e passes on e2e/. Re-run later via mise with the pinned tools: lint:licenses passes, the crewai and langgraph wrapper tests pass, and the protocol-compliance suite at CI's pin gives 46 pass / 0 fail / 6 skip (same as main).

Update (second commit): this tightens the fallback rule after review. An accumulated_usage holding only total_tokens previously returned zeros without checking usage. There is a new test case for this; it failed before the change and passes after. In goose's own write path (record_usage_metrics, v1.46.0), one UPDATE sets both objects from the same per-turn usage, so a field that is null in accumulated_usage is also null in usage. That means this case most likely comes only from imported or externally written sessions.

How I verified the goose shapes

I read goose's source at the release tags. I did not run goose.

• v1.37.0 and v1.38.0: Session has flat input_tokens, accumulated_input_tokens, and similar fields.
• v1.39.0 through v1.49.0: Session has usage: Usage and accumulated_usage: Usage with #[serde(default)]. Usage has five Option<i32> fields and no skip_serializing_if, so missing values come out as null. goose session export --format json is serde_json::to_string_pretty(&session).
• The doc comment saying input_tokens includes cache has been in token_usage.rs since 1.39.0.

Not verified

• I did not run a real goose binary, and no real goose session export was captured for this PR. The 1.46-shaped fixture uses the numbers from the issue, and its structure comes from goose source.
• I did not run the lifecycle e2e suite (mise run test:e2e), because it needs real goose and entire.
• I did not add protocol-compliance coverage in external-agents-tests. The JSON output shape is unchanged.
• No Entire-Checkpoint trailer (see the note at the top).
• fromOffset is still ignored, as before. goose only reports session-level totals.